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ABSTRACT 


This paper presents an ongoing development process for an experimental application that aims to train an inexperienced 
user to operate a mobile device and receive Augmented Reality (AR) guidelines for basic pick-and-place operations. The 
purpose of the presented work is to offer a learning environment for placing multiple physical tools in specific places, as 
indicated by various guideline notes. The current phase represents the 9’th experiment from a series of others (24) that are 
conducted in order to identify the proper specifications, challenges and development estimations for a training station 
(room) built for manufacturing industry workers. The application is developed from a gamified perspective allowing 
users to explore 2 different games. According to the rules of each game, the user advances to the next level if he manages 
to place each object in a corresponding placeholder until all the spaces from a 2X3 grid are occupied correctly. The 
process is timed and the user sees a custom countdown timer alongside with guiding messages and feedback. While most 
of technical challenges were successfully surpassed, the interaction together with the performance measuring are 
currently going through an improvement process meant to identify and assess the educational advantages offered by this 
type of solutions. 
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1. INTRODUCTION 


As a relatively recent enhancement added to the virtual reality (VR) technology evolution, AR brings a 
totally different level of interaction. The applicability of AR is focused on a variety of areas, starting from 
most common ones like gaming, assistive e-health, e-learning, to other more complex use-cases like military 
applications, smart city, industry and many others as presented in (Marr from Forbes, 2018). In order to bring 
the power of AR closer to the user and contribute to far-reaching scenarios, the most important factor is to 
combine AR with various other technologies that offer flexibility and mobility while allowing multiple ways 
of exploring AR visualizations as presented by (Aaltonen and Lehikoinen, 2006). 

The current paper presents an ongoing progress of an experimental application that uses AR capabilities 
of modern smartphones. The application (currently in alpha version) requires a smartphone with an Android 
operating system and AR support in order to use the detection and display features of the device for 
educational and training purposes. The presented test scenario uses a gamified experience in order to guide 
the user to recognize and place correctly several common bricolage tools into specific indicated placeholders. 
The guided toolbox is part of a series of experiments that will be used in the development of a 
self-configuring and dynamic industrial training module to overcome the current industry workforce 
challenges. Every experiment has a sole purpose of identifying the challenges behind a specific type of user 
interaction. The current use-case aims to understand how fast human workers can execute easy indications 
received through an AR-enabled mobile device in order to correctly execute dynamic pick-and place 
instructions. 
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2. EXPERIMENTAL APPROACH 


2.1 Scenario Description 


The scene setup configuration was inspired by the medical use-cases where there are several tools on 
different tables and the actors (doctors or nurses) are operating with them. Also, the mixed reality 
environment design has taken into consideration the conclusions described in (Antoniou et al, 2017). 

In this particular use-case (tool selection), the user is directed to pick a correct tool from a tool library 
(tool table). All the objects from the tool library were scanned before with Vuforia Object Scanner and a 
database stores the Object Data. There is a direct one to one correspondence between the tool library 
(software) and the tool table (physical object). Once an object is upheld, it has to be placed in the indicated 
spot on the placeholder table. The placeholder table is a physical table that has a one to one direct 
correspondence with a digital AR matrix (2X3) so that the system can process real time object tracking 
operations. 

All the user actions are sequential, pre-established and without completing a certain task, the user cannot 
proceed further in the process. The placeholder has 6 boxes, as displayed in Figure 1, each one of them 
represented by a colorful pattern, very different one from another in order to favor the precision of the 
solution at this stage. 


Figure 2. Placeholder table with 6 boxes and blue band 
strips for object placing direction. 
View from the Android device screenshot 


Figure 1. Placeholder table with 6 boxes as displayed on the 
working table 
View from the top. Camera shot. 


For guiding the correct tool placement, each box contains a colorful band strip (permanent AR object) 
that shows the correct direction for placing a tool. Without the proper tool direction, for some cases (marked 
with black stripes), even if the object would be placed in the right box the result would be considered 
incorrect. The color of the strip was chosen to be dark blue because of the big contrast with the background. 
The strip placement is affected by the distance and the tilting angle of the smartphone and it was one of the 
first signs of correct mapping, similar with the stabilizing methods described in (Andreasson, 2017). The 
empty placeholder with the blue direction strips are displayed in Figure 2. 


2.2 Gamified Experience 


In order to build a gamified experience, the whole process was designed around 2 available games: 

e the game of numbers: the rule is to place an object in a placeholder that’s indicated by a number 
(object x to box n mapping). The boxes are numbered in natural order. 

e the game of colours: the rule is to place an object in a placeholder that’s indicated by a color (object 
x to color c mapping). This game requires more attention since the placeholder images are very colorful, 
there is no order of the colors and multiple boxes can have the same color; 


142 


15th International Conference Mobile Learning 2019 


For both games, each attempt is presented like a challenge against time with a countdown timer displayed 
directly from the beginning. In case a user doesn’t put all the tools in the indicated boxes, after 120 seconds 
the game is interrupted. Every time a user makes a move, no mather if the move was correct or not, relevant 
instructions and feedback messages are displayed in order to guide him for the next step until he manages to 
place all the objects, as shown in Figures 3 and 4. 


Figure 4. Incomplete board, the 120 seconds have passed 


Figure 3. Completed board with all the 6 tools placed and not all the tools are in the correct position 


correctly and within the 120 seconds timeframe 


2.3 Scenes 


Unity allows the creation of multi scene applications. Each scene can have its own isolated environment that 
can contain components of several types. A scene can contain basic elements like buttons, pop-up messages 
and virtual objects, or more complex ones like animations and manipulations. As described in (Kim et al, 
2014), this particular feature allowed us to split the application in two scenes: START (only with basic 
elements) and GAME (mostly with complex elements). 

The START scene as displayed in Figure 5, has elements from the interface that are visible on the 
phone’s screen while running the application. In this scene a menu is displayed which has 3 native buttons 
and displays messages depending on what button you press. Through the menu the user can select what type 
of game they want to play. 

The GAME scene, as displayed in Figure 6, represents the game’s environment and has interface 
elements, buttons and text messages Vuforia-specific elements, that allows the use of virtual reality, AR 
Camera functions that operate with the scanned objects (Object Target) and the recorded images (Image 
Target). 


Figure 5. Start Scene as Displayed in the Unity View Figine Ream Peon asp daa he Unie Vier 


143 


ISBN: 978-989-8533-86-9 © 2019 


3. EARLY PERFORMANCE ANALYSIS 


During the development process, the application performance was tested on several devices in different 
environments and their performance was rated below as displayed in Table 1. 


Table 1. Performance Rating for Devices used during the Development Process 


Camera Resolution (mp) Operating System Rating (1-5) 
LG Nexus 5X Android 8.0 
Samsung Galaxy 52016 8 «dn 5. 


ZTE Blade V8 13+2 (dual) Android: 7.0 


Xiaomi Redmi Note 5 12+5 (dual) Android: 8.1 
Google Pinel XL Android 9.0 


The testing stage also concluded that there are 3 main factors that influence the overall performance: 
e lens camera quality (not the resolution) - device feature - directly impacts detection speed; 


@ reflection of light - environment feature - multiple tests were conducted in several different rooms 
and when the artificial light was very strong the objects were glossy, and the detection precision 
decreased drastically; 

e complexity of the item - object feature - at this stage, in order to increase the accuracy of scanned 
objects we decided to use only the items that have a large number of Vuforia Object Scanner Points 
(more than 100). A surface scan for a drilling machine is shown in Figure 7. 


Figure 7. Surface of the Objects and Number of Vuforia Object Scanner Points for a Drilling Machine 


For the testing stages of the alpha version, after the development was considered completed, it was 
decided to use the top 6 objects that had more scanner points (Adjustable Wrench, Allen Keys, Cordless 
Drill, Measuring Tape, Digital Caliper, Pliers), a laboratory with warm LED light (the natural light was 
shaded with translucent curtains) and the Google Pixel XL device (very good camera lens quality and 
superior Android version available). 


4. RESULTS AND CONCLUSION 


This paper presented an experimental application that follows an interactive method for learning 
pick-and-place operations using mobile AR technology. Intentionally, the operations were designed to be as 
simple as possible for the alpha version in order to focus on the following development tasks 

e precise detection for smaller and medium size objects; 
game logic with custom difficulty countdown timer; 
validation of every step actions; 
basic user interface, messages and real time feedback; 


° 
® 
® 
e performance optimizations when the number of objects on the board was increasing. 
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From a development perspective, the combination between Vuforia, Unity and Android represented a 
really challenging technical task, but under controlled light conditions and by using the proper mobile 
devices, the alpha version behaved unexpectedly good. 

Up to this stage, the current experiment is considered to be successful and very relevant for the bigger 
purpose (training station progress). During the development stage and after the first alpha version was 
finalized, the application was tested by more than 20 people of different ages, but with the same technical 
background (computer science students, professors, researchers). All the subjects were able to complete both 
games in less than 5 attempts. Most of the problems and delays were related to the usability of the mobile 
phone with AR capabilities and once the users understood the technology philosophy everything became 
clear. This outcome shows that the users can be trained using the described method and probably they can 
easily adapt without difficulties to more complex scenarios developed under the same logic. 

The overall performance of the application (object detection speed and precision) is highly influenced by 
the performance of the device and especially by the quality of the camera. The future extended study will 
also include detailed and explained performance measurements and recommendations for minimum system 
requirements, similar with the evaluation in (Fernandes, Cota, Moreira, 2015) and (Grahn, 2017). 

Until now, most of the technical challenges were identified and solved. Despite that, there are still two 
issues that remain unsolved and without solutions: 

e colorful scenes without human faces are considered complex by the internal Android 
autofocus feature and it would be very important to be able to control (force) the camera 
autofocus in order to speed up the placeholder detection 

e detection of smaller objects using the zoom feature or by approaching the smartphone to 
the target area 


The application is currently improved by adding the following changes and features: 
e@ expand the grid to a more complex physical board (6*3); 
detect multiple objects at the same time and ignore an object after it was correctly placed 
interchangeable images to allow various camera efficiency measurements; 
remote features as experimented in (Mourtzisa, Zogopoulosa and Vlachoua, 2017); 
gradual complexity levels to ensure a more natural learning process; 
multiplayer features for collaborative tasks; 
Microsoft Hololens AR glasses support, as experimented in (Heinonen, 2018). 
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